<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.9.4"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>Flow: net_flow/detail/low_lvl_io.hpp Source File</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr id="projectrow">
  <td id="projectalign">
   <div id="projectname">Flow<span id="projectnumber">&#160;2.0.0</span>
   </div>
   <div id="projectbrief">Flow project: Full implementation reference.</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.4 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
var searchBox = new SearchBox("searchBox", "search",'Search','.html');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
$(function() {
  initMenu('',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
/* @license-end */
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="dir_460afd45d0ec637b1427e0e06a6fbcf7.html">net_flow</a></li><li class="navelem"><a class="el" href="dir_6dd0f5e913f2b13fdf6025047ac045ce.html">detail</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle"><div class="title">low_lvl_io.hpp</div></div>
</div><!--header-->
<div class="contents">
<a href="low__lvl__io_8hpp.html">Go to the documentation of this file.</a><div class="fragment"><div class="line"><a id="l00001" name="l00001"></a><span class="lineno">    1</span><span class="comment">/* Flow</span></div>
<div class="line"><a id="l00002" name="l00002"></a><span class="lineno">    2</span><span class="comment"> * Copyright 2023 Akamai Technologies, Inc.</span></div>
<div class="line"><a id="l00003" name="l00003"></a><span class="lineno">    3</span><span class="comment"> *</span></div>
<div class="line"><a id="l00004" name="l00004"></a><span class="lineno">    4</span><span class="comment"> * Licensed under the Apache License, Version 2.0 (the</span></div>
<div class="line"><a id="l00005" name="l00005"></a><span class="lineno">    5</span><span class="comment"> * &quot;License&quot;); you may not use this file except in</span></div>
<div class="line"><a id="l00006" name="l00006"></a><span class="lineno">    6</span><span class="comment"> * compliance with the License.  You may obtain a copy</span></div>
<div class="line"><a id="l00007" name="l00007"></a><span class="lineno">    7</span><span class="comment"> * of the License at</span></div>
<div class="line"><a id="l00008" name="l00008"></a><span class="lineno">    8</span><span class="comment"> *</span></div>
<div class="line"><a id="l00009" name="l00009"></a><span class="lineno">    9</span><span class="comment"> *   https://www.apache.org/licenses/LICENSE-2.0</span></div>
<div class="line"><a id="l00010" name="l00010"></a><span class="lineno">   10</span><span class="comment"> *</span></div>
<div class="line"><a id="l00011" name="l00011"></a><span class="lineno">   11</span><span class="comment"> * Unless required by applicable law or agreed to in</span></div>
<div class="line"><a id="l00012" name="l00012"></a><span class="lineno">   12</span><span class="comment"> * writing, software distributed under the License is</span></div>
<div class="line"><a id="l00013" name="l00013"></a><span class="lineno">   13</span><span class="comment"> * distributed on an &quot;AS IS&quot; BASIS, WITHOUT WARRANTIES OR</span></div>
<div class="line"><a id="l00014" name="l00014"></a><span class="lineno">   14</span><span class="comment"> * CONDITIONS OF ANY KIND, either express or implied.</span></div>
<div class="line"><a id="l00015" name="l00015"></a><span class="lineno">   15</span><span class="comment"> * See the License for the specific language governing</span></div>
<div class="line"><a id="l00016" name="l00016"></a><span class="lineno">   16</span><span class="comment"> * permissions and limitations under the License. */</span></div>
<div class="line"><a id="l00017" name="l00017"></a><span class="lineno">   17</span><span class="comment"></span> </div>
<div class="line"><a id="l00018" name="l00018"></a><span class="lineno">   18</span><span class="comment">/// @file</span></div>
<div class="line"><a id="l00019" name="l00019"></a><span class="lineno">   19</span><span class="comment"></span><span class="preprocessor">#pragma once</span></div>
<div class="line"><a id="l00020" name="l00020"></a><span class="lineno">   20</span> </div>
<div class="line"><a id="l00021" name="l00021"></a><span class="lineno">   21</span><span class="preprocessor">#include &quot;<a class="code" href="detail_2net__flow__fwd_8hpp.html">flow/net_flow/detail/net_flow_fwd.hpp</a>&quot;</span></div>
<div class="line"><a id="l00022" name="l00022"></a><span class="lineno">   22</span><span class="preprocessor">#include &quot;<a class="code" href="low__lvl__packet_8hpp.html">flow/net_flow/detail/low_lvl_packet.hpp</a>&quot;</span></div>
<div class="line"><a id="l00023" name="l00023"></a><span class="lineno">   23</span><span class="preprocessor">#include &quot;<a class="code" href="linked__hash__set_8hpp.html">flow/util/linked_hash_set.hpp</a>&quot;</span></div>
<div class="line"><a id="l00024" name="l00024"></a><span class="lineno">   24</span> </div>
<div class="line"><a id="l00025" name="l00025"></a><span class="lineno">   25</span><span class="keyword">namespace </span><a class="code hl_namespace" href="namespaceflow_1_1net__flow.html">flow::net_flow</a></div>
<div class="line"><a id="l00026" name="l00026"></a><span class="lineno">   26</span>{</div>
<div class="line"><a id="l00027" name="l00027"></a><span class="lineno">   27</span><span class="comment"></span> </div>
<div class="line"><a id="l00028" name="l00028"></a><span class="lineno">   28</span><span class="comment">/**</span></div>
<div class="line"><a id="l00029" name="l00029"></a><span class="lineno">   29</span><span class="comment"> * The current outgoing packet pacing state, including queue of low-level packets to be sent, for a</span></div>
<div class="line"><a id="l00030" name="l00030"></a><span class="lineno">   30</span><span class="comment"> * given Peer_socket.</span></div>
<div class="line"><a id="l00031" name="l00031"></a><span class="lineno">   31</span><span class="comment"> *</span></div>
<div class="line"><a id="l00032" name="l00032"></a><span class="lineno">   32</span><span class="comment"> * This structure is a data store and not an object.  It is a separate `struct`, instead of directly</span></div>
<div class="line"><a id="l00033" name="l00033"></a><span class="lineno">   33</span><span class="comment"> * inside Peer_socket, to make the complex Peer_socket structure&#39;s definition shorter and easier to</span></div>
<div class="line"><a id="l00034" name="l00034"></a><span class="lineno">   34</span><span class="comment"> * understand.  Accordingly the Node code dealing with pacing (and working on this structure and</span></div>
<div class="line"><a id="l00035" name="l00035"></a><span class="lineno">   35</span><span class="comment"> * related data) is in a separate file (low_lvl_io.cpp instead of peer_socket.cpp or</span></div>
<div class="line"><a id="l00036" name="l00036"></a><span class="lineno">   36</span><span class="comment"> * node.cpp).</span></div>
<div class="line"><a id="l00037" name="l00037"></a><span class="lineno">   37</span><span class="comment"> *</span></div>
<div class="line"><a id="l00038" name="l00038"></a><span class="lineno">   38</span><span class="comment"> * ### Background on packet pacing ###</span></div>
<div class="line"><a id="l00039" name="l00039"></a><span class="lineno">   39</span><span class="comment"> * Empirically we&#39;ve seen that sending out many UDP datagrams at the same</span></div>
<div class="line"><a id="l00040" name="l00040"></a><span class="lineno">   40</span><span class="comment"> * time may cause extra -- sometimes quite major -- loss somewhere in the network.  To combat that,</span></div>
<div class="line"><a id="l00041" name="l00041"></a><span class="lineno">   41</span><span class="comment"> * we&#39;ve seen simple packet pacing be effective (and I hear Linux kernel does it too).  The idea is</span></div>
<div class="line"><a id="l00042" name="l00042"></a><span class="lineno">   42</span><span class="comment"> * to keep datagrams being sent at below a certain dynamically adjusting rate.  More specifically,</span></div>
<div class="line"><a id="l00043" name="l00043"></a><span class="lineno">   43</span><span class="comment"> * if we want to (and, subject to the send window, are allowed to) send N equally sized available</span></div>
<div class="line"><a id="l00044" name="l00044"></a><span class="lineno">   44</span><span class="comment"> * packets over T msec, it may prevent loss to send them equally spaced T/N msec apart, as</span></div>
<div class="line"><a id="l00045" name="l00045"></a><span class="lineno">   45</span><span class="comment"> * opposed to sending a burst of N packets immediately.  In other words the total send rate over</span></div>
<div class="line"><a id="l00046" name="l00046"></a><span class="lineno">   46</span><span class="comment"> * time period T is the same in either case, but in the latter case this is accomplished smoothly</span></div>
<div class="line"><a id="l00047" name="l00047"></a><span class="lineno">   47</span><span class="comment"> * instead of burstily.  This is pacing: minimize burstiness without lowering overall throughput.</span></div>
<div class="line"><a id="l00048" name="l00048"></a><span class="lineno">   48</span><span class="comment"> * The pending packets that cannot be sent immediately are added to a queue to be sent later.</span></div>
<div class="line"><a id="l00049" name="l00049"></a><span class="lineno">   49</span><span class="comment"> *</span></div>
<div class="line"><a id="l00050" name="l00050"></a><span class="lineno">   50</span><span class="comment"> * Suppose we keep the pacing system as above; begin a pacing time slice of length T, and over that time</span></div>
<div class="line"><a id="l00051" name="l00051"></a><span class="lineno">   51</span><span class="comment"> * period send packets spaced at T/N msec apart; once T is over, recompute T and N and repeat.  Let T be</span></div>
<div class="line"><a id="l00052" name="l00052"></a><span class="lineno">   52</span><span class="comment"> * decided.  We shouldn&#39;t use N = [number of packets pending to send], since at any moment many more can come</span></div>
<div class="line"><a id="l00053" name="l00053"></a><span class="lineno">   53</span><span class="comment"> * in and get queued (causing backup of the queue).  The natural solution is to use T = round trip time (SRTT)</span></div>
<div class="line"><a id="l00054" name="l00054"></a><span class="lineno">   54</span><span class="comment"> * and N = congestion window (CWND), since those by definition set the maximum rate we will allow packets to</span></div>
<div class="line"><a id="l00055" name="l00055"></a><span class="lineno">   55</span><span class="comment"> * be sent out, because the network cannot support more.  Thus, the interval between packets is S = SRTT /</span></div>
<div class="line"><a id="l00056" name="l00056"></a><span class="lineno">   56</span><span class="comment"> * CWND.</span></div>
<div class="line"><a id="l00057" name="l00057"></a><span class="lineno">   57</span><span class="comment"> *</span></div>
<div class="line"><a id="l00058" name="l00058"></a><span class="lineno">   58</span><span class="comment"> * To make this system more responsive, S = SRTT / CWND can be recomputed after queued packet is</span></div>
<div class="line"><a id="l00059" name="l00059"></a><span class="lineno">   59</span><span class="comment"> * sent (instead of recomputing it every SRTT as mentioned above; as SRTT and CWND are continuously</span></div>
<div class="line"><a id="l00060" name="l00060"></a><span class="lineno">   60</span><span class="comment"> * computed as acknowledgments come in, much more frequently than every SRTT).</span></div>
<div class="line"><a id="l00061" name="l00061"></a><span class="lineno">   61</span><span class="comment"> *</span></div>
<div class="line"><a id="l00062" name="l00062"></a><span class="lineno">   62</span><span class="comment"> * More precisely, whenever we send a packet, start a timer for S = SRTT / CWND msec.  While the</span></div>
<div class="line"><a id="l00063" name="l00063"></a><span class="lineno">   63</span><span class="comment"> * timer is running any new packet available for sending should be added to the packet queue Q and</span></div>
<div class="line"><a id="l00064" name="l00064"></a><span class="lineno">   64</span><span class="comment"> * not sent.  Once tomer fires, send packet at top of Q, if any (and start the timer again,</span></div>
<div class="line"><a id="l00065" name="l00065"></a><span class="lineno">   65</span><span class="comment"> * since packet was sent).  If Q is empty, and the timer is not running, and a new packet is</span></div>
<div class="line"><a id="l00066" name="l00066"></a><span class="lineno">   66</span><span class="comment"> * available for sending, it should be sent immediately.  Finally, any packet available for sending</span></div>
<div class="line"><a id="l00067" name="l00067"></a><span class="lineno">   67</span><span class="comment"> * until SRTT is known (i.e., before the first acknowledgment comes in) should be sent immediately</span></div>
<div class="line"><a id="l00068" name="l00068"></a><span class="lineno">   68</span><span class="comment"> * (all packets after that are paced per above algorithm).</span></div>
<div class="line"><a id="l00069" name="l00069"></a><span class="lineno">   69</span><span class="comment"> *</span></div>
<div class="line"><a id="l00070" name="l00070"></a><span class="lineno">   70</span><span class="comment"> * There is a technical limitation that complicates the above.  The timer period S can easily be</span></div>
<div class="line"><a id="l00071" name="l00071"></a><span class="lineno">   71</span><span class="comment"> * quite low, e.g., S = 100 msec / 100 = 1 msec.  In the current implementation (boost.asio) there</span></div>
<div class="line"><a id="l00072" name="l00072"></a><span class="lineno">   72</span><span class="comment"> * is no timer mechanism with that precision (platform-dependent, but see comment on util::Timer alias)</span></div>
<div class="line"><a id="l00073" name="l00073"></a><span class="lineno">   73</span><span class="comment"> * (we can *measure* time with much better precision, but we cannot *schedule* actions).  Therefore we</span></div>
<div class="line"><a id="l00074" name="l00074"></a><span class="lineno">   74</span><span class="comment"> * cannot always schedule timer for S.  Suppose the timer has a minimum scheduling precision of &lt; R.</span></div>
<div class="line"><a id="l00075" name="l00075"></a><span class="lineno">   75</span><span class="comment"> * Since we mathematically cannot avoid burstiness if S = SRTT / CWND &lt; R, we must minimize it;</span></div>
<div class="line"><a id="l00076" name="l00076"></a><span class="lineno">   76</span><span class="comment"> * namely, in that case, set S = R and, instead of 1 packet max during that period,</span></div>
<div class="line"><a id="l00077" name="l00077"></a><span class="lineno">   77</span><span class="comment"> * allow N = R / (SRTT / CWND) packets during that period.  Thus the simple &quot;queue packet each time</span></div>
<div class="line"><a id="l00078" name="l00078"></a><span class="lineno">   78</span><span class="comment"> * it is available, if timer is running&quot; policy changes to the slightly more complex &quot;start counter</span></div>
<div class="line"><a id="l00079" name="l00079"></a><span class="lineno">   79</span><span class="comment"> * K = N when timer starts; send packet and decrement K each time packet is available, until K is</span></div>
<div class="line"><a id="l00080" name="l00080"></a><span class="lineno">   80</span><span class="comment"> * down to zero, then queue packet.&quot;  Note that the latter reduces to the former when S &gt;= R (i.e.,</span></div>
<div class="line"><a id="l00081" name="l00081"></a><span class="lineno">   81</span><span class="comment"> * timer precision is not a problem).</span></div>
<div class="line"><a id="l00082" name="l00082"></a><span class="lineno">   82</span><span class="comment"> *</span></div>
<div class="line"><a id="l00083" name="l00083"></a><span class="lineno">   83</span><span class="comment"> * Now to formally define the invariants guiding the algorithm: let [T, T + S) be a time slice,</span></div>
<div class="line"><a id="l00084" name="l00084"></a><span class="lineno">   84</span><span class="comment"> * where T is the time point at which the first packet is sent once an SRTT is known.  S =</span></div>
<div class="line"><a id="l00085" name="l00085"></a><span class="lineno">   85</span><span class="comment"> * max(floor(SRTT / CWND), R), where SRTT and CWND (which is in units of max-block-sizes) are</span></div>
<div class="line"><a id="l00086" name="l00086"></a><span class="lineno">   86</span><span class="comment"> * determined at time T.  Each packet should be sent out as soon as possible after it becomes</span></div>
<div class="line"><a id="l00087" name="l00087"></a><span class="lineno">   87</span><span class="comment"> * available for sending, subject to the following constraint: at most N = S / (SRTT / CWND)</span></div>
<div class="line"><a id="l00088" name="l00088"></a><span class="lineno">   88</span><span class="comment"> * full-sized blocks&#39; worth of packets may be sent during the time slice [T, T + S); any packets</span></div>
<div class="line"><a id="l00089" name="l00089"></a><span class="lineno">   89</span><span class="comment"> * beyond that should be queued on packet queue Q (and sent based on the policy in his sentence, in</span></div>
<div class="line"><a id="l00090" name="l00090"></a><span class="lineno">   90</span><span class="comment"> * order of Q).  After the time slice is finished, the next time slice [T, T + S) is re-defined</span></div>
<div class="line"><a id="l00091" name="l00091"></a><span class="lineno">   91</span><span class="comment"> * at the time T the first packet available for sending appears (at which time S and N are</span></div>
<div class="line"><a id="l00092" name="l00092"></a><span class="lineno">   92</span><span class="comment"> * recomputed), and the above policy applies.  This continues indefinitely.</span></div>
<div class="line"><a id="l00093" name="l00093"></a><span class="lineno">   93</span><span class="comment"> *</span></div>
<div class="line"><a id="l00094" name="l00094"></a><span class="lineno">   94</span><span class="comment"> * ### Packets other than DATA ###</span></div>
<div class="line"><a id="l00095" name="l00095"></a><span class="lineno">   95</span><span class="comment"> * There is one final consideration.  The above discussion implicitly</span></div>
<div class="line"><a id="l00096" name="l00096"></a><span class="lineno">   96</span><span class="comment"> * treats all packets as DATA packets.  Since SYN/SYN_ACK/SYN_ACK_ACK packets all occur before the</span></div>
<div class="line"><a id="l00097" name="l00097"></a><span class="lineno">   97</span><span class="comment"> * first SRTT measurement is available (but see below to-do), they are not relevant.  However, there</span></div>
<div class="line"><a id="l00098" name="l00098"></a><span class="lineno">   98</span><span class="comment"> * are also ACK and RST packets.  First, let&#39;s deal with RST; an RST is by definition the very last</span></div>
<div class="line"><a id="l00099" name="l00099"></a><span class="lineno">   99</span><span class="comment"> * packet we can send on a connection.  We have no desire for it to be dropped any more than a DATA</span></div>
<div class="line"><a id="l00100" name="l00100"></a><span class="lineno">  100</span><span class="comment"> * packet; so it can be subject to pacing just like DATA and placed onto Q, etc.  Since it&#39;s the</span></div>
<div class="line"><a id="l00101" name="l00101"></a><span class="lineno">  101</span><span class="comment"> * last packet to be sent, there is nothing more to discuss.  However, that would mean the sending</span></div>
<div class="line"><a id="l00102" name="l00102"></a><span class="lineno">  102</span><span class="comment"> * of RST may occur after socket is in state CLOSED.  While we could make that work, it introduces</span></div>
<div class="line"><a id="l00103" name="l00103"></a><span class="lineno">  103</span><span class="comment"> * unnecessary complication; we prefer to consider the Peer_socket quite dead once it is in CLOSED.</span></div>
<div class="line"><a id="l00104" name="l00104"></a><span class="lineno">  104</span><span class="comment"> * Therefore I choose to not pace RSTs: send it immediately.  This MAY put the RST ahead of other</span></div>
<div class="line"><a id="l00105" name="l00105"></a><span class="lineno">  105</span><span class="comment"> * queued packets (ACK, DATA), but since RST indicates a sudden connection break, this is more or less</span></div>
<div class="line"><a id="l00106" name="l00106"></a><span class="lineno">  106</span><span class="comment"> * acceptable (and those other packets will not be sent, as we stop sending paced packets once in</span></div>
<div class="line"><a id="l00107" name="l00107"></a><span class="lineno">  107</span><span class="comment"> * CLOSED).  It may also increase the chance the RST is dropped, but I highly doubt it&#39;s significant</span></div>
<div class="line"><a id="l00108" name="l00108"></a><span class="lineno">  108</span><span class="comment"> * in reality.</span></div>
<div class="line"><a id="l00109" name="l00109"></a><span class="lineno">  109</span><span class="comment"> *</span></div>
<div class="line"><a id="l00110" name="l00110"></a><span class="lineno">  110</span><span class="comment"> * What about ACKs?  Probably we must place them onto Q, since sending them earlier would at least</span></div>
<div class="line"><a id="l00111" name="l00111"></a><span class="lineno">  111</span><span class="comment"> * mean changing the original order of Node&#39;s intended outgoing packet stream.  However, how many</span></div>
<div class="line"><a id="l00112" name="l00112"></a><span class="lineno">  112</span><span class="comment"> * bytes, in comparison to (N * max-block-size), is an ACK worth?  We could do this a couple of</span></div>
<div class="line"><a id="l00113" name="l00113"></a><span class="lineno">  113</span><span class="comment"> * ways.  It could be worth max-block-size, like a typical DATA packet; this is the conservative</span></div>
<div class="line"><a id="l00114" name="l00114"></a><span class="lineno">  114</span><span class="comment"> * choice.  However, it&#39;s probably not too realistic, since an ACK is much smaller than a full DATA.</span></div>
<div class="line"><a id="l00115" name="l00115"></a><span class="lineno">  115</span><span class="comment"> * It could be worth 0 bytes (i.e., send any ACK as soon as it&#39;s at the head of Q, regardless of</span></div>
<div class="line"><a id="l00116" name="l00116"></a><span class="lineno">  116</span><span class="comment"> * time slice), which is closer to the truth but not true (the other way).  Or it could be worth ~K</span></div>
<div class="line"><a id="l00117" name="l00117"></a><span class="lineno">  117</span><span class="comment"> * bytes, where K is the approximate size of its payload (not counting the common header data).  That</span></div>
<div class="line"><a id="l00118" name="l00118"></a><span class="lineno">  118</span><span class="comment"> * is most realistic, but would throw off a bit the appealing exact math when all DATA packets are</span></div>
<div class="line"><a id="l00119" name="l00119"></a><span class="lineno">  119</span><span class="comment"> * full-sized.  What to choose?  The answer is probably not too important, because in most</span></div>
<div class="line"><a id="l00120" name="l00120"></a><span class="lineno">  120</span><span class="comment"> * applications there will be no significant mixing of outgoing DATA and ACK packets (since either</span></div>
<div class="line"><a id="l00121" name="l00121"></a><span class="lineno">  121</span><span class="comment"> * one side or the other side is usually sending, not both simultaneously).  I choose to go with ACK</span></div>
<div class="line"><a id="l00122" name="l00122"></a><span class="lineno">  122</span><span class="comment"> * = 0 bytes, since it is fairly close to realistic yet also simple to implement.</span></div>
<div class="line"><a id="l00123" name="l00123"></a><span class="lineno">  123</span><span class="comment"> *</span></div>
<div class="line"><a id="l00124" name="l00124"></a><span class="lineno">  124</span><span class="comment"> * ### Thread safety ###</span></div>
<div class="line"><a id="l00125" name="l00125"></a><span class="lineno">  125</span><span class="comment"> * Meant to be accessed from thread W only.</span></div>
<div class="line"><a id="l00126" name="l00126"></a><span class="lineno">  126</span><span class="comment"> *</span></div>
<div class="line"><a id="l00127" name="l00127"></a><span class="lineno">  127</span><span class="comment"> * @todo One way to provide finer pacing, in the situation where we must send more than 1 packet</span></div>
<div class="line"><a id="l00128" name="l00128"></a><span class="lineno">  128</span><span class="comment"> * within the minimal schedulable timer period, is to rely on events other than timer clock ticks</span></div>
<div class="line"><a id="l00129" name="l00129"></a><span class="lineno">  129</span><span class="comment"> * that occur more frequently.  Namely, ACKs will possibly arrive faster that the minimal timer</span></div>
<div class="line"><a id="l00130" name="l00130"></a><span class="lineno">  130</span><span class="comment"> * schedulable ticks do.  Therefore, when a time slice begins, we can send no packets (instead of</span></div>
<div class="line"><a id="l00131" name="l00131"></a><span class="lineno">  131</span><span class="comment"> * all the packets that must be sent in that time slice, as is the case currently).  Then over the</span></div>
<div class="line"><a id="l00132" name="l00132"></a><span class="lineno">  132</span><span class="comment"> * course of the time slice we can opportunistically -- upon each ACK send the proper number of</span></div>
<div class="line"><a id="l00133" name="l00133"></a><span class="lineno">  133</span><span class="comment"> * queued packets up to that point into the time slice (this can be computed accurately, because we</span></div>
<div class="line"><a id="l00134" name="l00134"></a><span class="lineno">  134</span><span class="comment"> * can MEASURE time very accurately -- just can&#39;t schedule things as accurately).  Finally, once the</span></div>
<div class="line"><a id="l00135" name="l00135"></a><span class="lineno">  135</span><span class="comment"> * time slice does expire, send off any packets still remaining for that slice.  This way we can</span></div>
<div class="line"><a id="l00136" name="l00136"></a><span class="lineno">  136</span><span class="comment"> * opportunistically provide finer pacing, where the ACK clock allows it.</span></div>
<div class="line"><a id="l00137" name="l00137"></a><span class="lineno">  137</span><span class="comment"> *</span></div>
<div class="line"><a id="l00138" name="l00138"></a><span class="lineno">  138</span><span class="comment"> * @todo Obtain the first RTT measurement based on SYN-SYN_ACK or SYN_ACK-SYN_ACK_ACK interval; then</span></div>
<div class="line"><a id="l00139" name="l00139"></a><span class="lineno">  139</span><span class="comment"> * SRTT will be available from the start, and packet pacing can be enabled with the very first DATA</span></div>
<div class="line"><a id="l00140" name="l00140"></a><span class="lineno">  140</span><span class="comment"> * packet, avoiding the initial (albeit small) burst of DATA packets.  This is pretty easy.</span></div>
<div class="line"><a id="l00141" name="l00141"></a><span class="lineno">  141</span><span class="comment"> *</span></div>
<div class="line"><a id="l00142" name="l00142"></a><span class="lineno">  142</span><span class="comment"> * @todo As noted above we perform packet pacing but currently choose to assign a value of</span></div>
<div class="line"><a id="l00143" name="l00143"></a><span class="lineno">  143</span><span class="comment"> * 0 bytes to an ACK.  That is, while we do preserve the order of DATA and ACK packets -- if</span></div>
<div class="line"><a id="l00144" name="l00144"></a><span class="lineno">  144</span><span class="comment"> * both happen to be in the outgoing stream -- we do not delay the sending of the ACK once it is</span></div>
<div class="line"><a id="l00145" name="l00145"></a><span class="lineno">  145</span><span class="comment"> * the next packet to be sent out.  However, even so, an ACK&#39;s sending may be delayed by the</span></div>
<div class="line"><a id="l00146" name="l00146"></a><span class="lineno">  146</span><span class="comment"> * pacing applied to DATA packets intermixed with it.  Therefore the ACK delay measurement we</span></div>
<div class="line"><a id="l00147" name="l00147"></a><span class="lineno">  147</span><span class="comment"> * take here may be incorrect (too low) in that case.  This can cause overestimated RTTs on the</span></div>
<div class="line"><a id="l00148" name="l00148"></a><span class="lineno">  148</span><span class="comment"> * sender&#39;s side.  The to-do is to correct the ACK delay value in a given ACK by adding the</span></div>
<div class="line"><a id="l00149" name="l00149"></a><span class="lineno">  149</span><span class="comment"> * pacing delay (if any) of the ACK to the individual ACK delays within it.  Conceptually this</span></div>
<div class="line"><a id="l00150" name="l00150"></a><span class="lineno">  150</span><span class="comment"> * is similar to the `m_sent_when` value being set when choosing to send a DATA packet and then</span></div>
<div class="line"><a id="l00151" name="l00151"></a><span class="lineno">  151</span><span class="comment"> * corrected in the pacing module later.  This to-do is not important until we in practice start</span></div>
<div class="line"><a id="l00152" name="l00152"></a><span class="lineno">  152</span><span class="comment"> * mixing sending and receiving at the application layer... but still -- it&#39;s worth knowing that</span></div>
<div class="line"><a id="l00153" name="l00153"></a><span class="lineno">  153</span><span class="comment"> * there is a design bug here.</span></div>
<div class="line"><a id="l00154" name="l00154"></a><span class="lineno">  154</span><span class="comment"> *</span></div>
<div class="line"><a id="l00155" name="l00155"></a><span class="lineno">  155</span><span class="comment"> * @todo My quick look into Google BBR, the recent advanced congestion control algorithm, suggests</span></div>
<div class="line"><a id="l00156" name="l00156"></a><span class="lineno">  156</span><span class="comment"> * that the pacing algorithm can be used as part of congestion control rather than something merely</span></div>
<div class="line"><a id="l00157" name="l00157"></a><span class="lineno">  157</span><span class="comment"> * applied after congestion control has made the CWND decision.  I believe the idea would be this:</span></div>
<div class="line"><a id="l00158" name="l00158"></a><span class="lineno">  158</span><span class="comment"> * If we can somehow very reliably determine the available bandwidth, which Google BBR does purport to do,</span></div>
<div class="line"><a id="l00159" name="l00159"></a><span class="lineno">  159</span><span class="comment"> * then instead of controlling send rate via CWND, instead one could apply it to this pacing module.</span></div>
<div class="line"><a id="l00160" name="l00160"></a><span class="lineno">  160</span><span class="comment"> * It is a slight modification of the algorithm described above: instead of the rate of sending being</span></div>
<div class="line"><a id="l00161" name="l00161"></a><span class="lineno">  161</span><span class="comment"> * equal to CWND/SRTT, make it equal to the bandwidth determined through Google BBR.  If I understand</span></div>
<div class="line"><a id="l00162" name="l00162"></a><span class="lineno">  162</span><span class="comment"> * correctly, Google BBR does maintain a CWND as well, but this is a safety upper bound, in case the</span></div>
<div class="line"><a id="l00163" name="l00163"></a><span class="lineno">  163</span><span class="comment"> * computed bandwidth is too high to be practical (in this case CWND will slow down data being delivered</span></div>
<div class="line"><a id="l00164" name="l00164"></a><span class="lineno">  164</span><span class="comment"> * to the pacing module -- but, normally, this would not be the case, and the pacing module would be,</span></div>
<div class="line"><a id="l00165" name="l00165"></a><span class="lineno">  165</span><span class="comment"> * in practice, the thing controlling the sending rate).  Note this is a pretty big conceptual change;</span></div>
<div class="line"><a id="l00166" name="l00166"></a><span class="lineno">  166</span><span class="comment"> * in particular, it would make the queue Q grow much larger than what we see currently.</span></div>
<div class="line"><a id="l00167" name="l00167"></a><span class="lineno">  167</span><span class="comment"> *</span></div>
<div class="line"><a id="l00168" name="l00168"></a><span class="lineno">  168</span><span class="comment"> * ### `Timer` vs util::schedule_task_from_now() ###</span></div>
<div class="line"><a id="l00169" name="l00169"></a><span class="lineno">  169</span><span class="comment"> * In other places we have tended to replace a `Timer` with the far simpler util::schedule_task_from_now() (etc.)</span></div>
<div class="line"><a id="l00170" name="l00170"></a><span class="lineno">  170</span><span class="comment"> * facility (which internally uses a `Timer` but hides its various annoyances and caveats).  Why not here?</span></div>
<div class="line"><a id="l00171" name="l00171"></a><span class="lineno">  171</span><span class="comment"> * Answer: These tasks are potentially scheduled *very* frequently; and the firing periods can push the limits of how</span></div>
<div class="line"><a id="l00172" name="l00172"></a><span class="lineno">  172</span><span class="comment"> * small they can effectively be.  Therefore, the perf caveats (as listed in util::schedule_task_from_now() doc header)</span></div>
<div class="line"><a id="l00173" name="l00173"></a><span class="lineno">  173</span><span class="comment"> * apply in a big way -- moreover, exactly when the effects would be felt the most (namely at high throughputs -- which</span></div>
<div class="line"><a id="l00174" name="l00174"></a><span class="lineno">  174</span><span class="comment"> * in turn is likeliest to peg process use).  We should definitely not risk that in this case.</span></div>
<div class="line"><a id="l00175" name="l00175"></a><span class="lineno">  175</span><span class="comment"> */</span></div>
<div class="line"><a id="l00176" name="l00176"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html">  176</a></span><span class="keyword">struct </span><a class="code hl_struct" href="structflow_1_1net__flow_1_1Send__pacing__data.html">Send_pacing_data</a> :</div>
<div class="line"><a id="l00177" name="l00177"></a><span class="lineno">  177</span>  <span class="keyword">private</span> boost::noncopyable</div>
<div class="line"><a id="l00178" name="l00178"></a><span class="lineno">  178</span>{</div>
<div class="line"><a id="l00179" name="l00179"></a><span class="lineno">  179</span>  <span class="comment">// Types.</span></div>
<div class="line"><a id="l00180" name="l00180"></a><span class="lineno">  180</span><span class="comment"></span> </div>
<div class="line"><a id="l00181" name="l00181"></a><span class="lineno">  181</span><span class="comment">  /// Short-hand for FIFO of low-level packets.  `queue&lt;&gt;` dumbly has no `clear()`, so just use `deque&lt;&gt;` directly.</span></div>
<div class="line"><a id="l00182" name="l00182"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">  182</a></span><span class="comment"></span>  <span class="keyword">using </span><a class="code hl_typedef" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">Packet_q</a> = std::deque&lt;Low_lvl_packet::Const_ptr&gt;;</div>
<div class="line"><a id="l00183" name="l00183"></a><span class="lineno">  183</span> </div>
<div class="line"><a id="l00184" name="l00184"></a><span class="lineno">  184</span>  <span class="comment">// Data.</span></div>
<div class="line"><a id="l00185" name="l00185"></a><span class="lineno">  185</span><span class="comment"></span> </div>
<div class="line"><a id="l00186" name="l00186"></a><span class="lineno">  186</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00187" name="l00187"></a><span class="lineno">  187</span><span class="comment">   * The time point at which the last pacing time slice began; or epoch if no packets sent so far</span></div>
<div class="line"><a id="l00188" name="l00188"></a><span class="lineno">  188</span><span class="comment">   * (i.e., no time slices yet =&gt; undefined).  The time period</span></div>
<div class="line"><a id="l00189" name="l00189"></a><span class="lineno">  189</span><span class="comment">   * [#m_slice_start, #m_slice_start + #m_slice_period) defines the range of time points in the time</span></div>
<div class="line"><a id="l00190" name="l00190"></a><span class="lineno">  190</span><span class="comment">   * slice.</span></div>
<div class="line"><a id="l00191" name="l00191"></a><span class="lineno">  191</span><span class="comment">   */</span></div>
<div class="line"><a id="l00192" name="l00192"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a">  192</a></span>  <a class="code hl_typedef" href="namespaceflow.html#a9d9cc2eeb10d398cff5591d446b763b8">Fine_time_pt</a> <a class="code hl_variable" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a">m_slice_start</a>;</div>
<div class="line"><a id="l00193" name="l00193"></a><span class="lineno">  193</span><span class="comment"></span> </div>
<div class="line"><a id="l00194" name="l00194"></a><span class="lineno">  194</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00195" name="l00195"></a><span class="lineno">  195</span><span class="comment">   * The length of the current pacing time slice period; this depends on congestion window and SRTT</span></div>
<div class="line"><a id="l00196" name="l00196"></a><span class="lineno">  196</span><span class="comment">   * on the containing socket, at the time point #m_slice_start.  Undefined (`zero()`) if #m_slice_start</span></div>
<div class="line"><a id="l00197" name="l00197"></a><span class="lineno">  197</span><span class="comment">   * is undefined.</span></div>
<div class="line"><a id="l00198" name="l00198"></a><span class="lineno">  198</span><span class="comment">   */</span></div>
<div class="line"><a id="l00199" name="l00199"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143">  199</a></span>  <a class="code hl_typedef" href="namespaceflow.html#a48799f1263cdeedec125be51a3db2b79">Fine_duration</a> <a class="code hl_variable" href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143">m_slice_period</a>;</div>
<div class="line"><a id="l00200" name="l00200"></a><span class="lineno">  200</span><span class="comment"></span> </div>
<div class="line"><a id="l00201" name="l00201"></a><span class="lineno">  201</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00202" name="l00202"></a><span class="lineno">  202</span><span class="comment">   * This many bytes worth of DATA packets may still be sent, at this time, within the</span></div>
<div class="line"><a id="l00203" name="l00203"></a><span class="lineno">  203</span><span class="comment">   * time slice defined by #m_slice_start and #m_slice_period; if this is less than the data size of</span></div>
<div class="line"><a id="l00204" name="l00204"></a><span class="lineno">  204</span><span class="comment">   * `m_packet_q.front()`, then no more can be sent, and any packets that need to be sent must be sent</span></div>
<div class="line"><a id="l00205" name="l00205"></a><span class="lineno">  205</span><span class="comment">   * in the next time slice (i.e., at time point `&gt;= m_slice_start + m_slice_period`).  Undefined</span></div>
<div class="line"><a id="l00206" name="l00206"></a><span class="lineno">  206</span><span class="comment">   * (zero) if #m_slice_start is undefined.</span></div>
<div class="line"><a id="l00207" name="l00207"></a><span class="lineno">  207</span><span class="comment">   *</span></div>
<div class="line"><a id="l00208" name="l00208"></a><span class="lineno">  208</span><span class="comment">   * Why track it in bytes instead of multiples of max-block-size (as in the `struct` doc header)?</span></div>
<div class="line"><a id="l00209" name="l00209"></a><span class="lineno">  209</span><span class="comment">   * Since the DATA packets in the queue may be of varying sizes, this allows us to more precisely</span></div>
<div class="line"><a id="l00210" name="l00210"></a><span class="lineno">  210</span><span class="comment">   * (proportionally) count them against the #m_bytes_allowed_this_slice budget.</span></div>
<div class="line"><a id="l00211" name="l00211"></a><span class="lineno">  211</span><span class="comment">   */</span></div>
<div class="line"><a id="l00212" name="l00212"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46">  212</a></span>  <span class="keywordtype">size_t</span> <a class="code hl_variable" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46">m_bytes_allowed_this_slice</a>;</div>
<div class="line"><a id="l00213" name="l00213"></a><span class="lineno">  213</span><span class="comment"></span> </div>
<div class="line"><a id="l00214" name="l00214"></a><span class="lineno">  214</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00215" name="l00215"></a><span class="lineno">  215</span><span class="comment">   * Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be</span></div>
<div class="line"><a id="l00216" name="l00216"></a><span class="lineno">  216</span><span class="comment">   * sent, with the head element the one to be sent next.  Each new pending packet is to be added at</span></div>
<div class="line"><a id="l00217" name="l00217"></a><span class="lineno">  217</span><span class="comment">   * the tail.  Invariant: `m_packet_q.front()`, if it exists, is of type DATA after any</span></div>
<div class="line"><a id="l00218" name="l00218"></a><span class="lineno">  218</span><span class="comment">   * Node::sock_pacing_new_packet_ready() or Node::sock_pacing_process_q() call returns.  (This is consistent</span></div>
<div class="line"><a id="l00219" name="l00219"></a><span class="lineno">  219</span><span class="comment">   * with the aforementioned &quot;do not re-order non-DATA packets, but do not pace them&quot; philosophy.)</span></div>
<div class="line"><a id="l00220" name="l00220"></a><span class="lineno">  220</span><span class="comment">   */</span></div>
<div class="line"><a id="l00221" name="l00221"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331">  221</a></span>  <a class="code hl_typedef" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">Packet_q</a> <a class="code hl_variable" href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331">m_packet_q</a>;</div>
<div class="line"><a id="l00222" name="l00222"></a><span class="lineno">  222</span><span class="comment"></span> </div>
<div class="line"><a id="l00223" name="l00223"></a><span class="lineno">  223</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00224" name="l00224"></a><span class="lineno">  224</span><span class="comment">   * When running, #m_packet_q is non-empty, #m_bytes_allowed_this_slice &lt; data size of</span></div>
<div class="line"><a id="l00225" name="l00225"></a><span class="lineno">  225</span><span class="comment">   * `m_packet_q.front()`, and the timer will fire when this time slice ends, and thus the head packet</span></div>
<div class="line"><a id="l00226" name="l00226"></a><span class="lineno">  226</span><span class="comment">   * in #m_packet_q must be sent.  In all other situations the timer is inactive.  (Note that if the</span></div>
<div class="line"><a id="l00227" name="l00227"></a><span class="lineno">  227</span><span class="comment">   * timer is not running, that does not mean any packet becoming available can be sent immediately;</span></div>
<div class="line"><a id="l00228" name="l00228"></a><span class="lineno">  228</span><span class="comment">   * if the queue is empty and `m_bytes_allowed_this_slice == N`, the timer is off, but if a packet</span></div>
<div class="line"><a id="l00229" name="l00229"></a><span class="lineno">  229</span><span class="comment">   * comes in with data size &gt; `N` before `m_slice_start + m_slice_period` [within the current time</span></div>
<div class="line"><a id="l00230" name="l00230"></a><span class="lineno">  230</span><span class="comment">   * slice], then #m_slice_timer must be set to fire at time `m_slice_start + m_slice_period`.)</span></div>
<div class="line"><a id="l00231" name="l00231"></a><span class="lineno">  231</span><span class="comment">   */</span></div>
<div class="line"><a id="l00232" name="l00232"></a><span class="lineno"><a class="line" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91">  232</a></span>  <a class="code hl_typedef" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6">util::Timer</a> <a class="code hl_variable" href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91">m_slice_timer</a>;</div>
<div class="line"><a id="l00233" name="l00233"></a><span class="lineno">  233</span> </div>
<div class="line"><a id="l00234" name="l00234"></a><span class="lineno">  234</span>  <span class="comment">// Constructors/destructor.</span></div>
<div class="line"><a id="l00235" name="l00235"></a><span class="lineno">  235</span><span class="comment"></span> </div>
<div class="line"><a id="l00236" name="l00236"></a><span class="lineno">  236</span><span class="comment">  /**</span></div>
<div class="line"><a id="l00237" name="l00237"></a><span class="lineno">  237</span><span class="comment">   * Initializes data to initial state (no active time slice).</span></div>
<div class="line"><a id="l00238" name="l00238"></a><span class="lineno">  238</span><span class="comment">   *</span></div>
<div class="line"><a id="l00239" name="l00239"></a><span class="lineno">  239</span><span class="comment">   * @param task_engine</span></div>
<div class="line"><a id="l00240" name="l00240"></a><span class="lineno">  240</span><span class="comment">   *        IO service for the timer stored as data member.</span></div>
<div class="line"><a id="l00241" name="l00241"></a><span class="lineno">  241</span><span class="comment">   */</span></div>
<div class="line"><a id="l00242" name="l00242"></a><span class="lineno">  242</span>  <span class="keyword">explicit</span> <a class="code hl_function" href="structflow_1_1net__flow_1_1Send__pacing__data.html#ab7f2dc947fc3c57a6a0e93efb33c8a08">Send_pacing_data</a>(<a class="code hl_typedef" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4">util::Task_engine</a>* task_engine);</div>
<div class="line"><a id="l00243" name="l00243"></a><span class="lineno">  243</span>}; <span class="comment">// struct Send_pacing_data</span></div>
<div class="line"><a id="l00244" name="l00244"></a><span class="lineno">  244</span> </div>
<div class="line"><a id="l00245" name="l00245"></a><span class="lineno">  245</span>} <span class="comment">// namespace flow::net_flow</span></div>
<div class="ttc" id="adetail_2net__flow__fwd_8hpp_html"><div class="ttname"><a href="detail_2net__flow__fwd_8hpp.html">net_flow_fwd.hpp</a></div></div>
<div class="ttc" id="alinked__hash__set_8hpp_html"><div class="ttname"><a href="linked__hash__set_8hpp.html">linked_hash_set.hpp</a></div></div>
<div class="ttc" id="alow__lvl__packet_8hpp_html"><div class="ttname"><a href="low__lvl__packet_8hpp.html">low_lvl_packet.hpp</a></div></div>
<div class="ttc" id="anamespaceflow_1_1net__flow_html"><div class="ttname"><a href="namespaceflow_1_1net__flow.html">flow::net_flow</a></div><div class="ttdoc">Flow module containing the API and implementation of the Flow network protocol, a TCP-inspired stream...</div><div class="ttdef"><b>Definition:</b> <a href="asio_2node_8cpp_source.html#l00024">node.cpp:25</a></div></div>
<div class="ttc" id="anamespaceflow_1_1util_html_ac3e89a8a271b0ddc76ac2a0ce488dea4"><div class="ttname"><a href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4">flow::util::Task_engine</a></div><div class="ttdeci">boost::asio::io_context Task_engine</div><div class="ttdoc">Short-hand for boost.asio event service, the central class of boost.asio.</div><div class="ttdef"><b>Definition:</b> <a href="util__fwd_8hpp_source.html#l00135">util_fwd.hpp:135</a></div></div>
<div class="ttc" id="anamespaceflow_1_1util_html_ae7416d64d2989051104bb396e28e15e6"><div class="ttname"><a href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6">flow::util::Timer</a></div><div class="ttdeci">boost::asio::basic_waitable_timer&lt; Fine_clock &gt; Timer</div><div class="ttdoc">boost.asio timer.</div><div class="ttdef"><b>Definition:</b> <a href="util__fwd_8hpp_source.html#l00202">util_fwd.hpp:202</a></div></div>
<div class="ttc" id="anamespaceflow_html_a48799f1263cdeedec125be51a3db2b79"><div class="ttname"><a href="namespaceflow.html#a48799f1263cdeedec125be51a3db2b79">flow::Fine_duration</a></div><div class="ttdeci">Fine_clock::duration Fine_duration</div><div class="ttdoc">A high-res time duration as computed from two Fine_time_pts.</div><div class="ttdef"><b>Definition:</b> <a href="common_8hpp_source.html#l00416">common.hpp:416</a></div></div>
<div class="ttc" id="anamespaceflow_html_a9d9cc2eeb10d398cff5591d446b763b8"><div class="ttname"><a href="namespaceflow.html#a9d9cc2eeb10d398cff5591d446b763b8">flow::Fine_time_pt</a></div><div class="ttdeci">Fine_clock::time_point Fine_time_pt</div><div class="ttdoc">A high-res time point as returned by Fine_clock::now() and suitable for precise time math in general.</div><div class="ttdef"><b>Definition:</b> <a href="common_8hpp_source.html#l00413">common.hpp:413</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html">flow::net_flow::Send_pacing_data</a></div><div class="ttdoc">The current outgoing packet pacing state, including queue of low-level packets to be sent,...</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00176">low_lvl_io.hpp:178</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_a100a406213a5edf2854440397719bb91"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#a100a406213a5edf2854440397719bb91">flow::net_flow::Send_pacing_data::m_slice_timer</a></div><div class="ttdeci">util::Timer m_slice_timer</div><div class="ttdoc">When running, m_packet_q is non-empty, m_bytes_allowed_this_slice &lt; data size of m_packet_q....</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00232">low_lvl_io.hpp:232</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_a9cfd632911c3c4acd8e048dae28ded46"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#a9cfd632911c3c4acd8e048dae28ded46">flow::net_flow::Send_pacing_data::m_bytes_allowed_this_slice</a></div><div class="ttdeci">size_t m_bytes_allowed_this_slice</div><div class="ttdoc">This many bytes worth of DATA packets may still be sent, at this time, within the time slice defined ...</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00212">low_lvl_io.hpp:212</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_ab7f2dc947fc3c57a6a0e93efb33c8a08"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#ab7f2dc947fc3c57a6a0e93efb33c8a08">flow::net_flow::Send_pacing_data::Send_pacing_data</a></div><div class="ttdeci">Send_pacing_data(util::Task_engine *task_engine)</div><div class="ttdoc">Initializes data to initial state (no active time slice).</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8cpp_source.html#l01048">low_lvl_io.cpp:1048</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_ae7b8c15eaf55d2891bf160574fd41739"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#ae7b8c15eaf55d2891bf160574fd41739">flow::net_flow::Send_pacing_data::Packet_q</a></div><div class="ttdeci">std::deque&lt; Low_lvl_packet::Const_ptr &gt; Packet_q</div><div class="ttdoc">Short-hand for FIFO of low-level packets. queue&lt;&gt; dumbly has no clear(), so just use deque&lt;&gt; directly...</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00182">low_lvl_io.hpp:182</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_aed68c9912e8c2ada5a9fcef0f9df2331"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#aed68c9912e8c2ada5a9fcef0f9df2331">flow::net_flow::Send_pacing_data::m_packet_q</a></div><div class="ttdeci">Packet_q m_packet_q</div><div class="ttdoc">Queue of low-level packets to be sent to the remote endpoint, in order in which they are to be sent,...</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00221">low_lvl_io.hpp:221</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_af0c881081c6706dba393ef814386b143"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#af0c881081c6706dba393ef814386b143">flow::net_flow::Send_pacing_data::m_slice_period</a></div><div class="ttdeci">Fine_duration m_slice_period</div><div class="ttdoc">The length of the current pacing time slice period; this depends on congestion window and SRTT on the...</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00199">low_lvl_io.hpp:199</a></div></div>
<div class="ttc" id="astructflow_1_1net__flow_1_1Send__pacing__data_html_af1e8f11a36db3b967d9a5f61b115750a"><div class="ttname"><a href="structflow_1_1net__flow_1_1Send__pacing__data.html#af1e8f11a36db3b967d9a5f61b115750a">flow::net_flow::Send_pacing_data::m_slice_start</a></div><div class="ttdeci">Fine_time_pt m_slice_start</div><div class="ttdoc">The time point at which the last pacing time slice began; or epoch if no packets sent so far (i....</div><div class="ttdef"><b>Definition:</b> <a href="low__lvl__io_8hpp_source.html#l00192">low_lvl_io.hpp:192</a></div></div>
</div><!-- fragment --></div><!-- contents -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Mar 28 2025 22:55:35 for Flow by&#160;<a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.4
</small></address>
</body>
</html>
